1. Field of the Invention
This invention relates to the field of computer and network interconnections, backplane busses and bus-to-bus bridges, and more specifically to a method, apparatus, and system for building a very high speed, fault tolerant, high-data-integrity network of bus-to-bus bridges. This network is completely transparent in operation to modules that use it to communicate with each other. Modules that are connected to different, interconnected busses communicate as if they were connected to the same bus. The invention provides a bandwidth many times that of a single bus and can be used to replace and/or extend conventional backplane busses or to form a network switch fabric.
2. Description of the Prior Art
A major issue in the information age is the speed at which data can be transferred between points. This issue exists in computers both for transferring data between memory and a central processing unit, and for transferring data between devices and/or memory. The issue also exists for transferring data between computers or digitized voice data between telephone units.
As processor speed and network traffic has increased, the physical limitations of traditional interconnects have become more apparent. With commonly available home computers operating at a clock speed of more than 200 MHz, the computing bottleneck is generally a result of moving data within the system and not as a result of processing the data. Rambuss technology is one approach that addresses a similar problem in providing a high bandwidth interconnection between a processor and memory. Other approaches exist for generalized high speed interconnects such as the scaleable coherent interface (SCI).
A problem is that vast amounts of data need to be transported from one place to another as quickly as possible with minimal latency and maximum throughput. This issue is complicated by the advantages of remaining backward compatible with existing device interface standards because of the existing market investment in such devices. Thus, a need exists for an interconnect that operates, and is configured, as a bus-to-bus bridge and can be used without changes to a computer""s operating system or drivers related to the functions of the interconnected device.
Historically, devices have been attached to a computer bus such as the Small Computer System Interconnect (SCSI) bus, or the Peripheral Component Interconnect (PCI) bus. These busses make physical tradeoffs between the bus bandwidth, the length of the bus, and cost. They are limited to only a single data transfer between devices on these busses at any given time.
Some bus topologies, such as a ring topology, allow for multiple transfers to occur between devices. In a ring topology, a number of nodes are connected together using a serial arrangement of point-to-point busses such that each node xe2x80x9cdirectlyxe2x80x9d communicates to a limited number of nodes. Data passes from one node to another by passing through intermediate nodes. Thus, with a four-node ring at least four separate data transfers can occur at the same time. Because data may pass through a number of nodes on the ring, the latency of each node must be minimized to achieve maximal bandwidth between two non-adjacent nodes. Often, such rings have a sufficiently complex protocol that software (or complex and therefore slow or expensive hardware) is required to process each unit of data (cell or packet) received by a node. This software increases the latency in each node.
SCI (IEEE std 1596-1992) can be used as a computer interconnect. This usage is well described in Multiprocess Interconnection using SCI, by Ivan Tving, (copyright)1994 and is included by reference as illustrative of the prior art. One problem with SCI is that it is not completely synchronous. The specification requires an xe2x80x9celastic bufferxe2x80x9d to serve as a synchronization barrier between one node and its adjacent downstream node. This elastic buffer is used to get around the problems of clock drift and bit skew between the parallel data signals. This pleisochronous operation requires that idle symbols must be added to the data stream. Inclusion of these idle symbols decreases the interconnect""s bandwidth, increases each SCI node""s latency and increases the cost of SCI. SCI also supports cache coherent operation (not required for many applications) increasing protocol complexity.
Additionally, details of the PCI bus operation are described in the PCI Local Bus Specification, revision 2.0, (copyright)1993, PCI Special Interest Group, that is included by reference as illustrative of the prior art. Further, details relating to how PCI busses are bridged together are provided by the PCI to PCI Bridge Architecture Specification, version 1.0, (copyright)1994, PCI Special Interest Group, that also is included by reference as illustrative of the prior art.
Because the interconnect is fundamental to devices that transport data, the device fails if the interconnect fails. Thus, there exists a need for a fault tolerant interconnection. Such a fault tolerant interconnection should provide a high data integrity interconnection, automatically detect and adjust for failures in the interconnect and allow for replacing nodes on the interconnect while the interconnect is operating (a hot swap capability).
An object of the present invention is to provide a system interconnect with nodes and paths that provides a frequency reference to each node on the interconnect, with the frequency reference being used to generate the same frequency node clock in each node and with data from one node being sent to another adjacent node without including a data clock signal.
Another object of the present invention is to provide an interconnect initialization mechanism and method that propagates the initialization from node to node on the interconnect.
Another object of the present invention is to provide a system interconnect that detects interrupt conditions on a bus attached to a node and of propagating the interrupt condition to another node on the system interconnect.
Another object of the present invention is to provide a system interconnect that performs flow control to limit the number of cells transmitted by a node.
Another object of the present invention is to provide a link selection register that selects which path of a dual ring network is used when transmitting a cell from one node to another.
Another object of the present invention is a multiple ring system interconnect that allows cells to be sent from one ring to another by using a routing tag in the cell.
Another object of the present invention is to provide a backup frequency reference mechanism to provide fault-tolerant operation of the system interconnect.
Another object of the present invention is to provide a system interconnect that communicates bus operations performed on one bus to be communicated to devices on another bus with the busses being connected to differing nodes on the interconnect.
A preferred embodiment of the present invention includes a method and apparatus for implementing a system interconnect for transporting a first cell containing a plurality of data between a plurality of nodes. The system interconnect includes a first unidirectional path between each pair of adjacent nodes of said plurality of nodes. The first unidirectional path includes an incoming end, an outgoing end, a first plurality of signal lines and a frequency reference line. The pair of adjacent nodes have a source node and a receiver node with each of the plurality of nodes including an input section connected to the incoming end of an input path and an output section connected to the outgoing end of an output path. A frequency reference signal is provided on the frequency reference line to the plurality of nodes. A phase lock loop frequency multiplier generates a node clock based on said frequency reference signal. The node clock has the same frequency for each of said plurality of nodes. The source node includes a source node clock and the receiver node includes a receiver node clock. The output section of the source node transmits the first cell over a first unidirectional path to the input section of the receiver node. This transmission is accomplished by emitting a plurality of data transitions, representing the first cell, onto a first plurality of signal lines at the outgoing end of the first unidirectional path. The input section of the receiver node receives the data transitions from the first plurality of signal lines at the incoming end of the first unidirectional path.
A method and apparatus for initializing a plurality of nodes on a ring network of a system interconnect includes a network also having a plurality of links with initialization being accomplished by a first node emitting a reset sequence to a second node and the second node further emits the reset sequence.
A method and apparatus for configuring a system interconnect for transporting a cell from a source node on a ring network to a destination node on said network includes a system interconnect having a plurality of nodes, a first path and a second path, with one of said plurality of nodes being a configuration node. Configuration is accomplished by initializing the nodes and with the configuration node emitting a first identifying address sequence containing a configuration node address onto a first path, a second node receiving from the first path the first identifying address sequence, the second node modifying the configuration node address to create a second node address and the second node emitting a second identifying address sequence containing the second node address onto the first path.
A method and apparatus for transporting a cell from a source node on a ring network, with a plurality of links, to a destination node of a plurality of destination nodes includes tracking a number of incomplete transactions and delaying the generation of the cell if one more than the number of incomplete transaction is outside a sliding window width. After one more than the number of incomplete transactions is within the sliding window width, the cell is generated and transported to the destination node.
A method and apparatus for transporting a cell from a source node on a ring network to a destination node on the network, includes a clockwise path and a counterclockwise path, with transporting the cell being accomplished by accessing a link selection register at the source node to select which path to use to transport the cell and then transporting the cell over the selected path.
Another embodiment of the method and apparatus for transporting a cell from a source node on a network to a destination node on the network wherein the network has a plurality of rings with each ring having a plurality of nodes and the plurality of rings have a source ring and a second ring connected by a ring coupler node, cell transportation is accomplished by the source node constructing a cell with a routing tag containing a first and second forward hop address and transmitting the cell onto the source ring. The ring coupler node swallows the cell from the source ring and transmits the cell onto the second ring.
Another embodiment is a method and apparatus for transporting a cell (with a routing tag) from a first ring to a second ring by a ring-to-ring coupler node that has an address wherein the transportation is accomplished by receiving the cell from the first ring, determining disposition of the cell solely from said routing tag and the address of the ring-to-ring coupler node and transmitting the cell onto the second ring.
In a method and apparatus that recovers from the loss of an initial frequency reference signal on a first path, recovery is accomplished by delaying a backup frequency reference signal to create a delayed frequency reference signal that is in phase with the initial frequency reference signal, and after the loss of the initial frequency reference signal is detected, using the delayed frequency reference signal.
A method and apparatus for accessing a first bus connected to a system interconnect at a first node, modules access the first bus by performing a bus operation on a second bus connected to the system interconnect at a second node and converting the bus operation into a cell. The cell is then transported over the system interconnect from the second node to the first node, and the first node then performs an equivalent bus operation on the first bus after receiving the cell.
Another embodiment is a method and apparatus for constructing a routing tag for a cell based on an address provided by a bus operation on a bus. Constructing the routing tag is accomplished by capturing the address from the bus and converting the address to a value stored in the routing tag.
An advantage of the present invention is that it provides a very high speed system interconnect with very little latency in each node on the interconnect without a distributed data clock.
Another advantage of the present invention is that it provides a high speed replacement for existing computer buses.
Another advantage of the present invention is that each node provides self adjusting flow control capabilities.
Another advantage of the present invention is that it provides a link selection register that selects which path of a multiple ring system interconnect to use to transmit a cell.
Another advantage of the present invention is that it provides a self contained routing tag that allows a cell to hop from ring to ring in a multiple ring system interconnect.
Another advantage of the present invention is that it provides a backup frequency reference for fault tolerant operation.
These and other objects and advantages of the present invention will no doubt become obvious to those of ordinary skill in the art after having read the following detailed description of the preferred embodiments that are illustrated in the various drawing figures.